1. Field of the Invention
The present invention relates to a neural network, and more specifically, it relates to a neural network which artificially simulates nerve cells of an organism by electric circuits.
2. Description of the Prior Art
In recent years, study has been deeply made to apply a neural network to artificially simulate nerve cells of an organism in the field of pattern recognition and artificial intelligence.
FIG. 1 shows nerve cell structure of an organism. Dendrites receive stimuli from axons, of other nerve cells. The magnitudes of stimuli transferred to the dendrites depend on the degrees of coupling of synapses. When the total magnitude of stimulus received by one nerve cell exceeds a predetermined value, this nerve cell is excited and transfers the stimulus to other nerve cells through the axons. In 1958, F. Rosenblatt advocated a perceptron, which implements the function of a pattern discriminator by modelling the nerve cell.
FIG. 2A is a conceptual diagram schematically showing the perceptron. The neural network shown in FIG. 2A simulates a state in which four nerve cells 101 to 104 are connected to a nerve cell 105 through synapses, w.sub.1, w.sub.2, w.sub.3 and w.sub.4. Assuming that x.sub.1 to x.sub.4 represent magnitudes of stimuli output from the nerve cells 101 to 104, the summation of stimulus input to the nerve cell 105 is ##EQU1## As shown in FIG. 2B, output y from the nerve cell 105 is "1" when the input stimulus quantity ##EQU2## is greater than a threshold value h, while the same is "0" when the input stimulus quantity ##EQU3## is less than the threshold value h.
The perceptron shown in FIG. 2A can program the degrees of coupling of connector elements so that the output y goes to "1" only when x.sub.1, x.sub.2, x.sub.3 and x.sub.4 have specific patterns. Thus, the perceptron can be applied to a character reader, for example, which encodes information of read characters into binary numbers. In this case, the degrees of coupling of the connector elements may be programmed so that the output y goes to "1" only when binary codes corresponding to desired characters are input to the perceptron. The feature of such a character reader implemented by the perceptron resides in that it can discriminate read characters even if the characters are incomplete due to blurring or chipping, by allowing for small errors.
The perceptron has two operating modes including a learning mode (see FIG. 2C) of programming values of the connector elements to attain required correspondence between input and output as hereinabove described, and an association mode (see FIG. 2D) of outputting y which corresponds to input x in response to the programmed correspondence.
In contrast to a general computer, there is no need to humanly develop an algorithm for pattern discrimination and input a program in a calculator in the learning mode. The input signal x and a desirable output signal t (educator signal) corresponding thereto may be inputted in the perceptron, so that the perceptron itself automatically changes the values of the connector elements so that the output y coincides with the educator signal t. This is apparently similar to learning of the brain of a human, and hence called a learning mode.
It is known that a synapse value w.sub.i may be changed to reduce the square error E=1/2(y-t).sup.2 of the output y and the expected value t** in order to make the learning of the neural network shown in FIG. 2A such that the output y from the nerve cell 105 goes to an expected value t when the nerve cell 101 to 104 output x.sub.1, x.sub.2, x.sub.3 and x.sub.4, for example, in this learning mode. When the synapse value w.sub.i is changed to w.sub.i +.DELTA.w.sub.i, the square error E is converted to ##EQU4## Therefore, assuming that ##EQU5## where E represents a positive constant ##EQU6## and hence the square error E is reduced. Thus a process of changing the synapse value w.sub.i to: ##EQU7## where E'=E f' and ##EQU8## may be repeated. The expression (1) is known as the learning rule of the perceptron. The method of changing x in accordance with ##EQU9## for reducing the function f with variable x is known as the steepest descent method.
In general, such a perceptron function has been implemented by software processing of a computer. Therefore, a computer has been generally required in order to use the perceptron, and hence the system has been complicated. Thus, there has been a demand to implement the perceptron technique by a simple hardware circuit.
Japanese Patent Laying-Open Gazette No. 81755/1984 discloses implementing functions of nerve cells and simulation of a learning operation through a technique different from the perceptron technique. FIG. 3 shows a simulation circuit disclosed in the said Gazette. Referring to FIG. 3, an artificial neuron has a plurality of external input terminals 202, which are connected with input lines I.sub.1, I.sub.2, . . . , I.sub.N from an external stimulus source such as another neuron or a sensor. The external input terminals 202 are connected with synapses S.sub.1, S.sub.2, . . . , S.sub.N respectively. Symbol + or - indicates whether the corresponding synapse is stimulative (+) or inhibitory (-).
When an input signal is received on one of the input lines from the external stimulus source, a corresponding synapse carries a four-bit number expressing its potential (hereinafter referred to as "synapse value") on a data bus 204 which is connected between the synapse and an output generator 205. if the synapse is stimulative, the synapse value is positive and is added to a time-changing value stored in the output generator 205. If the synapse is inhibitory, on the other hand, the synapse value is subtracted from the value stored in the generator. If the synapse value. is positive and the value in the generator thereby exceeds a predetermined threshold value, the output generator 205 outputs pulses outputted from one or more neurons to an output line 206. The output pulses from the neurons may be employed to control a driver (in a step motor, for example). Alternatively, these pulses may be supplied to synapses of another neuron as input signals, to be further integrated. The output pulses are fed back to the respective synapses of the original neuron through a line 207, to control change in the synapse values. The simulation circuit shown in FIG. 3 further has two input terminals 208 and 209, which are shown as positive and negative fixers.
Assuming that an input signal is received in the synapse S.sub.2, which is stimulative, for example, from the external stimulus source and thereafter the output generator 205 derives an output signal, reduction of the synapse value is started from the time of generation of the output signal. Input in the output generator 205 is also changed in response to the synapse value, while a positive fixer signal is generated when the output from the output generator 205 reaches a desired value and input in the input terminal 208, thereby to fix the value of the synapse S.sub.2.
This learning operation, i.e., the process of correcting the synapse value, is adapted to count down the synapse value from the initial value and fix the same when the output from the output generator 205 reaches the desirable value, by exhaustively scanning the synapse value and optimizing the same. Thus, this method is inferior in learning efficiency. On the other hand, the aforementioned perceptron technique of optimizing the synapse value by the steepest descent method is considered to be superior in learning efficiency to the simulation circuit shown in FIG. 3. This is because the steepest descent method is adapted to determine whether the output is greater or less than the desired value on the basis of a judgement as to whether a partial differential coefficient of the square error is positive or negative thereby to reduce the synapse value when the output is greater than the desired value while increasing the synapse value when the output is less than the desired value, and hence the synapse value converges to the optimum value at a speed higher than that in the circuit shown in FIG. 3.
As hereinabove described, the conventional perceptron requires a computer for implementation thereof, and hence the system is complicated. On the other hand, the conventional hardware simulation circuit shown in FIG. 3 is inferior in learning efficiency, although it requires no computer.